Part One: Research Question¶

Section A: Do customers enrolled in more service offerings display lower levels of churn?¶

The dataset for this project consists of AnyTelecom Company's observations on customers and customer churn. As the project's data analyst, my task is to prepare the data for explatory data analysis and ultimately answer the following research question: Do customers enrolled in more service offerings display lower levels of churn?

Relevance¶

Churn (the measure of customers who stop using a product or service) can have a significant impact on a company's business performance. It is in AnyTelecom Company's best interest to examine the potential relationship between the number of service offerings a customer is enrolled in and the rate of churn.

Section B: Data Dictionary¶

Part Two: Data Cleaning Plan¶

Section C.1: Techniques & Steps¶

Techniques¶

The quality of the data will be assessed using the following techniques:

• Duplicate Detection
• Missing Value Detection
• Outlier Detection

Duplicate Rows¶

The pandas.DataFrame.duplicated() function will be applied on the dataset to identify 1) duplication across entire records and 2) duplication across Customer_id values.

Missing Values¶

The pandas.DataFrame.isnull.sum() function will be applied to identify the number of missing values in each column of the dataset. Columns that are found to contain missing values will be visualized using the missingno.matrix() function.

Outliers¶

Outliers will be identified in two stages. First, boxplots will be used to visually identify outliers for relevant numeric variables in the dataset; these boxplots will be produced using the pandas.DataFrame.boxplot() function. Next, the number of outliers in each feature will be calculated by applying the scipy.stats.zscore() function on each column of the dataset.

Section C.2: Justification of Methods¶

Duplicate Rows¶

The pandas.DataFrame.duplicated() function returns a Series that describes which records in the dataset are duplicated and which records are not (pandas, 2023). If a record is found to be duplicated, these results can be queried to determine the record's index in order to drop the row from the dataset.

Missing Values¶

The pandas.DataFrame.isnull.sum() function returns the sum of missing values for each column in the dataset (pandas, 2023). The missingno.matrix() function provides a visualization of the distribution of missing values within a given column (Larose & Larose, 2019).

Outliers¶

Boxplots are used to represent the spread of data and allow one to visually assess the presence of outliers in a sample. The pandas.DataFrame.boxplot() function can be used to produce boxplots for numerical data in the dataset (pandas, 2023).

In statistics, z-scores describe the numeric relationship between a value and the mean. When combined with the standard cutoff value (+/-3), z-scores can also determine if a value is an outlier (Larose & Larose, 2019). The scipy.stats.zscore() function computes the z-score of each value in a sample (The SciPy Community, 2023).

Section C.3: Programming Environment¶

Programming Language¶

Python will be used to clean the dataset. Python is an open-source, robust programming language that includes several libraries and packages used to perform data analytics, data science, data engineering, and machine learning.

Libraries¶

The following libraries/packages will be used in conjunction with Python:

• pandas (data manipulation and analysis)
• missingno.matrix (visualization of missing values)
• scipy.stats (z-score computation)
• numpy.random (random number generation)
• scikit-learn.preprocessing (data standardization)
• scikit-learn.decomposition (data reduction)
• matplotlib.pyplot (plotting and visualization)

Integrated Development Environment¶

JupyterLab will be used as the Integrated Development Environment. JupyterLab is a web-based, open-source data science tool that allows users to write and share code in the form of a notebook.

Section C.4: Code¶

Out[4]:

(10000, 52)

Out[5]:

<class 'pandas.core.frame.DataFrame'>

RangeIndex: 10000 entries, 0 to 9999

Data columns (total 52 columns):

 #   Column                Non-Null Count  Dtype  

---  ------                --------------  -----  

 0   Unnamed: 0            10000 non-null  int64  

 1   CaseOrder             10000 non-null  int64  

 2   Customer_id           10000 non-null  object 

 3   Interaction           10000 non-null  object 

 4   City                  10000 non-null  object 

 5   State                 10000 non-null  object 

 6   County                10000 non-null  object 

 7   Zip                   10000 non-null  int64  

 8   Lat                   10000 non-null  float64

 9   Lng                   10000 non-null  float64

 10  Population            10000 non-null  int64  

 11  Area                  10000 non-null  object 

 12  Timezone              10000 non-null  object 

 13  Job                   10000 non-null  object 

 14  Children              7505 non-null   float64

 15  Age                   7525 non-null   float64

 16  Education             10000 non-null  object 

 17  Employment            10000 non-null  object 

 18  Income                7510 non-null   float64

 19  Marital               10000 non-null  object 

 20  Gender                10000 non-null  object 

 21  Churn                 10000 non-null  object 

 22  Outage_sec_perweek    10000 non-null  float64

 23  Email                 10000 non-null  int64  

 24  Contacts              10000 non-null  int64  

 25  Yearly_equip_failure  10000 non-null  int64  

 26  Techie                7523 non-null   object 

 27  Contract              10000 non-null  object 

 28  Port_modem            10000 non-null  object 

 29  Tablet                10000 non-null  object 

 30  InternetService       10000 non-null  object 

 31  Phone                 8974 non-null   object 

 32  Multiple              10000 non-null  object 

 33  OnlineSecurity        10000 non-null  object 

 34  OnlineBackup          10000 non-null  object 

 35  DeviceProtection      10000 non-null  object 

 36  TechSupport           9009 non-null   object 

 37  StreamingTV           10000 non-null  object 

 38  StreamingMovies       10000 non-null  object 

 39  PaperlessBilling      10000 non-null  object 

 40  PaymentMethod         10000 non-null  object 

 41  Tenure                9069 non-null   float64

 42  MonthlyCharge         10000 non-null  float64

 43  Bandwidth_GB_Year     8979 non-null   float64

 44  item1                 10000 non-null  int64  

 45  item2                 10000 non-null  int64  

 46  item3                 10000 non-null  int64  

 47  item4                 10000 non-null  int64  

 48  item5                 10000 non-null  int64  

 49  item6                 10000 non-null  int64  

 50  item7                 10000 non-null  int64  

 51  item8                 10000 non-null  int64  

dtypes: float64(9), int64(15), object(28)

memory usage: 4.0+ MB

Duplicated Rows¶

Out[7]:

Out[8]:

Missing Values¶

Out[9]:

Unnamed: 0                 0

CaseOrder                  0

Customer_id                0

Interaction                0

City                       0

State                      0

County                     0

Zip                        0

Lat                        0

Lng                        0

Population                 0

Area                       0

Timezone                   0

Job                        0

Children                2495

Age                     2475

Education                  0

Employment                 0

Income                  2490

Marital                    0

Gender                     0

Churn                      0

Outage_sec_perweek         0

Email                      0

Contacts                   0

Yearly_equip_failure       0

Techie                  2477

Contract                   0

Port_modem                 0

Tablet                     0

InternetService            0

Phone                   1026

Multiple                   0

OnlineSecurity             0

OnlineBackup               0

DeviceProtection           0

TechSupport              991

StreamingTV                0

StreamingMovies            0

PaperlessBilling           0

PaymentMethod              0

Tenure                   931

MonthlyCharge              0

Bandwidth_GB_Year       1021

item1                      0

item2                      0

item3                      0

item4                      0

item5                      0

item6                      0

item7                      0

item8                      0

dtype: int64

Out[12]:

Children             2495

Age                  2475

Income               2490

Techie               2477

Phone                1026

TechSupport           991

Tenure                931

Bandwidth_GB_Year    1021

dtype: int64

Out[13]:

<Axes: >

Out[14]:

array([[<Axes: title={'center': 'Children'}>,

        <Axes: title={'center': 'Age'}>,

        <Axes: title={'center': 'Income'}>,

        <Axes: title={'center': 'Tenure'}>,

        <Axes: title={'center': 'Bandwidth_GB_Year'}>]], dtype=object)

Outliers¶

Out[15]:

Out[16]:

<Axes: >

Out[17]:

<Axes: >

Out[18]:

<Axes: >

Out[19]:

<Axes: >

Out[20]:

<Axes: >

Out[21]:

<Axes: >

Out[22]:

<Axes: >

Out[23]:

<Axes: >

Out[24]:

<Axes: >

Out[25]:

<Axes: >

Out[28]:

Out[29]:

144

Out[30]:

110

Out[31]:

491

Out[32]:

12

Out[33]:

165

Out[34]:

94

Out[35]:

3

Part Three: Data Cleaning¶

Section D.1: Data Quality Findings¶

Duplicates¶

No duplicates were found.

Missing Values¶

Missing values were found for the following features:

• Children (2495)
• Age (2475)
• Income (2490)
• Techie (2477)
• Phone (1026)
• TechSupport (991)
• Tenure (931)
• Bandwidth_GB_Year (1021)

Outliers¶

The following features contained outliers:

• Children (144)
• Income (110)
• Outage_sec_perweek (491)
• Email (12)
• Contacts (165)
• Yearly_equip_failure (94)
• MonthlyCharge (3)

Section D.2: Methods¶

Duplicates¶

Since no duplicates were found, it was not necessary to implement methods to handle duplicate values.

Missing Values¶

Depending on the data type and characteristics of the data, different methods were used to handle missing values.

Qualitative Data¶

Features of the qualitative data type were:

• Techie
• Phone
• TechSupport

As it is not possible to compute the mean or median on categorical data, missing values for these features were imputed using the mode (Larose & Larose, 2019). To do this, the pandas.DataFrame.fillna() function was applied on the Techie, Phone, and TechSupport columns. The pandas.DataFrame.fillna() function is used to fill NA/NaN values with user-specified values. In this case, NA/NaN values were replaced using the mode of each feature. To calculate the mode, the pandas.Series.mode() function was applied on each column (pandas, 2023).

Quantitative Data: Uniform Distribution¶

Age was the only uniform feature of the quantitative data type. To preserve the shape of this feature, missing values were imputed using random values. This data cleaning step was performed in several phases. First, the numpy.random.randint() function was used to create an array of random integers within the range of Age. Next, a new DataFrame was created; this DataFrame contained the records where the Age variable was null. Finally, the random array of integers was inserted into the new DataFrame, and the values of the DataFrame were used to replace NA/NaN values in Age by applying the pandas.DataFrame.fillna() function on the column.

Quantitative Data: Skewed & Bi-Modal Distribution¶

Features of the quantitative data type were:

• Children
• Income
• Tenure
• Bandwidth_GB_Year

To avoid further skewing the distribution of these features, the median, rather than the mean, was used to impute missing values for these features (Larose & Larose, 2019). The pandas.DataFrame.fillna() function was applied on the Children, Income, Tenure, and Bandwidth_GB_Year columns. NA/NaN values were replaced using the median of each feature, and this value was calculated by applying the pandas.Series.median() function on each column.

Outliers¶

If there was indication that a set of outliers were the result of error or illegitimate entry, records containing these outliers were removed from the dataset (Larose & Larose, 2019). Otherwise, outliers were retained.

Indication of Illegitimacy¶

For this project, illegitimacy was determined based on the volume of outliers found in a given feature. It was assumed that if a feature contained a sparse number of outliers, then those outliers were illegitimate. In Section C.4: Outliers, queries were performed to identify the number of outliers in each feature.

Illegitimate Outliers¶

The features MonthlyCharge and Email contained relatively few outliers (3 and 12, respectively). Records containing these outliers were removed from the dataset. To do this, the pandas.DataFrame.drop() function was applied on the MonthlyCharge and Email columns for records that contained outliers.

Legitimate Outliers¶

Each of the following features contained more than 100 outliers:

• Children
• Income
• Outage_sec_perweek
• Email
• Contacts
• Yearly_equip_failure
• MonthlyCharge

These features were retained in the dataset.

Section D.3: Code¶

Handling Missing Values¶

Out[37]:

Techie         0

Phone          0

TechSupport    0

dtype: int64

Out[38]:

count    7525.000000

mean       53.275748

std        20.753928

min        18.000000

25%        35.000000

50%        53.000000

75%        71.000000

max        89.000000

Name: Age, dtype: float64

Out[41]:

0

Out[42]:

<Axes: >

Out[44]:

Children             0

Income               0

Tenure               0

Bandwidth_GB_Year    0

dtype: int64

Out[45]:

array([[<Axes: title={'center': 'Children'}>,

        <Axes: title={'center': 'Income'}>],

       [<Axes: title={'center': 'Tenure'}>,

        <Axes: title={'center': 'Bandwidth_GB_Year'}>]], dtype=object)

Out[46]:

<Axes: >

Handling Outliers¶

Out[48]:

<Axes: >

Out[49]:

<Axes: >

Re-Expression of Categorical Variables¶